Title: AWK is great - fail2ban list
Date: 2017-09-23 14:00
Category: CLI
Tags: CLI, awk, fail2ban, regex


###All hail AWK
It took me a while to get around to teaching myself [awk], but since learning it some years ago I have not looked back.
There is quite a lot that you can do with it; while sometimes a full program (python/ruby/etc) might be more robust...
There is something very satisfying about perfecting an awk 'oneliner'.
<sup>(I may stretch some peoples' definition of oneliner)</sup>
While I say 'awk' in this article, I use [gawk] as it has some nice things added.


####regex is also great
If you haven't taken the time to learn [regex], and some of it's more advanced features, you are missing out.
It's flexiblity to handle different matches is amazing, and once you start using [capture groups] you will wonder why no one told you sooner.


#####Minor variations on a theme
You will want to note that there is a minor bit of variation between implementations of regex.
<sup>(though nowhere near as bad as markdown fragmentation)</sup>
This mostly consists newer implementations having a few more features bolted on, that are followed in other yet newer implementations -- you just will miss them in older programs like `sed`.
One example would be what is referred to as `perl regex`, which adds some nice escape codes like: `\d`, `\D`, `\w`, `\W`, `\s`, `\S`, etc.

Here `\d` is a quick way to specify digit, though you can always use the old school way instead: `[0-9]`.
With other added escape codes the old school way gets more tedious, so being able to use `\w` for word character is very convenient -- and `\W` for any NOT word character.


###Today's sample
Niro, a friend on the interwebz, was asking for a convenient way of listing all IP's currently blocked by [fail2ban] while only listing a unique IP a single time.

My initial google fu [turned up]:

    :::bash
    fail2ban-client status | grep "Jail list:" | sed "s/ //g" | awk '{split($2,a,",");for(i in a) system("fail2ban-client status " a[i])}' | grep "Status\|IP list"

This did list each jail's banned IPs...
Though it also printed cruff for jails without current bans, cruff around the IPs, and didn't limit output to unique IPs.
It also failed to use awk to its full potential. \***points at the `grep` to `sed` to `awk` to... `grep`**\*


####awk remix
So, lets turn that awk up to 11:

    :::bash
    fail2ban-client status | awk '/Jail list/ {match($0, /^.*\:\s+(.*)/, m1); split(m1[1], s, ", "); for(i in s){while("fail2ban-client status " s[i] |& getline var){match(var, /IP list\:\s+(.*)/, m2); if(m2[1] != ""){h=h m2[1]}}}; gsub(/ /, "\n", h); print h | "sort -u"}'

**NOTE:** This is using `gawk` for the fancy `match(){}` function.

Here is the same thing broken up into a more readable format:

    :::php
    fail2ban-client status |                                            ## Get raw list of jails
        awk '/Jail list/                                                ## Get single relivant line
            {
            match($0, /^.*\:\s+(.*)/, m1);                              ## Grab just the jails
            split(m1[1], s, ", ");                                      ## Make jails into usable format
            for(i in s)                                                 ## Iterate over jails
                {
                while("fail2ban-client status " s[i] |& getline var)    ## Get raw list of IPs for each jail
                    {
                    match(var, /IP list\:\s+(.*)/, m2);                 ## Grab just the IPs for each jail
                    if(m2[1] != "")                                     ## Weed out non matches
                        {
                        h=h m2[1]                                       ## Append and merge IPs to list
                        }
                    }
                };
            gsub(/ /, "\n", h);                                         ## Separate IPs with newline
            print h | "sort -u"                                         ## sort, uniq, and print IPs one per line
            }'


###TLDR;
You should learn awk and regex. :)


###Links
[awk]  
[gawk]  
[regex]  
[capture groups]  
[fail2ban]  


[awk]: http://www.grymoire.com/Unix/Awk.html
[gawk]: https://www.gnu.org/software/gawk/manual/gawk.html
[regex]: https://www.gnu.org/software/grep/manual/html_node/Regular-Expressions.html
[capture groups]: http://www.rexegg.com/regex-capture.html
[fail2ban]: https://www.fail2ban.org/
[turned up]: https://askubuntu.com/a/893108
